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Purpose  of  Overall  Effort 


Develop  a  methodology  to  define  enterprise-wide 
measures  that  reflect  the  “health”  of  a  government 
organization  that  supports  acquisition. 


Apply  methodology  to  ensure  alignment  between  the 
enterprise-level  goals  of  an  organization  and  the 
measures  used  to  characterize  that  organization's 
performance. 


Use  these  measures  as  a  guide  to  their  overall 
performance  and  performance  improvement  effort. 
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Overview  Outline 

Methodology 

Major  components 

•  BSC 

•  GQ(I)M 

Example  use 

•  Initial  measurement  areas 

•  Indicators 

Summary 
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Mission 


Ciarify  mission  & 
Vision  statement 


Deveiop 
Strategic  Goais 


Vision 


Strategic  Goais 


Sub-Goais 


Balanced  Scorecard 


Financial 
•  Sub-Goals 


Methodology 

Overview 


Map  Sub-Goals  to  each 
quadrant  of  the 
Balanced  Score  Card 


Customer 

L  •  Sub-Goals 

Internal  Business 
•  Sub-Goals 

Learning  &  Growth 
*  Sub-Goals 

For  each  BSC  Quadrant 


Apply  GQ(I)M  to: 

-  identify  measurement  areas 

-  develop  measurement  goals 

-  pose  relevant  questions 

-  postulate  indicators 

-  identify  data  elements 


Indicators 


iIl  IIit: 


Data  Elements 
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Major  Components 

GQ(I)M 

•  Align  measures  with  goals;  ensure  measures  selected 
will  be  used 


Balanced  Scorecard 

•  Ensure  set  of  measures  provides  coverage  of  all 
elements  of  performance;  avoid  hidden  trade-offs 

Process  Model  of  Performance 

•  Select  measures  that  are  most  meaningful  with  respect 
to  selected  areas  of  performance;  prefer  outcome  then 
output  measures  over  process  and  input  measures 
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A  Balanced  Scorecard  Perspective 
on  Performance 


metrics,  Ratheon 
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Success  Vs  Progress  Indicators 


Goal 

T" 


strategy  to 
accomplish 
the  goal 


Analysis  Indicators  ^ 


Success 

Criteria 


Success  Indicators 


% 

1  2  3  4  1  2  3  4 

Reporting  Periods 

Progress  Indicators 
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Identifying  Potentiai  Measures:  A 
Process  Modei  of  Performance 


Inputs 


Potentiai 

Resources 

Throughput, 

Products 

Impact  on 

Measures 

consumed 

tasks 

and  services 

customer  or  user 
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Identifying  Potentiai  Measures:  A 
Process  Modei  of  Performance 


Inputs 


Goal:  Increase  Customer  Satisfaction 


Inputs  -  dollars 
spent  on  customer 
service  training, 
dollars  spent  on 
quality  assurance 


Process  -  number  of 
work  product 
inspections 
performed,  number 
of  tests  performed 


Outputs  -  number  of 
new  features 
released,  resolution 
time  for  customer 
service  calls 


Outcomes:  trends  in 
customer 

satisfaction  survey 
data,  number  of 
defects  reported 
after  release 
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Balanced  Scorecard  Perspective: 

A  Multi-dimensional  view 


Financial  Perspective 


Customer  Perspective 


I 


Vision 

And 

Strategy 


Innovation  and  Learning 
Perspective 


Internal  Business  Perspective 


► 


Source:  Kaplan  and  Norton,  ’’Putting  the  Balanced  Scorecard  to  Work”  Harvard  Business  Review,  Sept-Oct  1993 
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Defining 

Indicators 

& 

Measures 

Based 

Upon 

Goais 


GOAL(s) 


SLOC  Staff-hours  Trouble  Reports  Milestone  dates 
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Overview  Outline 

Methodology 

Major  components 

•  BSC 

•  GQ(I)M 

Example  use 

•  Initial  measurement  areas 

•  Indicators 

Summary 
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Organization 


Example  based  on  aggregate  of  several  organizations  with 
similar  characteristics 

•  Government  agency  consisting  of  300  management, 
administrative,  and  technical  personnel 

•  Development,  maintenance  and  enhancement  of 
system  components  of  fielded  systems,  and  acquisition 
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Use  of  Methodology  -  Example 


Mission 


Develop,  acquirer,  and 
maintain  integrated 
software-intensive 
systems 


Measurement  Workshop 
Purpose 

-  Clarify  Mission  and  Vision 

-  Develop  Strategic  Goals 

-  Derive  Sub-Goals 

-  Map  sub-goals  to  each 
quadrant  of  the  BSC 


Strategic  Goals 


Sub-Goals 


Internal 

Financial 

Customer 

Learnina  and 

Business 

•  Funding 

•  Timeliness 

Growth 

•  Quality 

stability 

•  Responsiveness 

•  Enhance  staff 

deficiencies 

•  Delivered 

•  Communication 

capability 

•  Availabie 

costs 

•  Relationship 

•  Improvement 

resources 

•  Etc. 

•  Quality  of  products 

quality 

•  Etc. 

•  Etc. 

•  Etc. 
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Success  Criteria 


Baianced  Scorecard 


BSC  Quadrant 

Strategic 

Sub-Goals 

Success 

Criteria 

Financial 

Internal  Business  Process 

Learning  and  Growth 

Customer 

©  2002  by  Carnegie  Mellon  University 


Carnegie  Mellon 

Software  Engineering  Institute 


Typical  Questions  Related  to  Sub-Goals 


Customers’  Viewpoint 

•  What  is  important  to  our  customer?  What  are  the 
customers’  “hot  buttons”? 

•  How  do  our  customers  evaluate  timeliness? 

•  What  does  the  customer  consider  a  quality  product? 
Are  there  any  standards  or  goals  currently  set  by  the 
customer? 

•  How  and  what  do  our  customers  currently  evaluate  our 
organization? 

•  Etc. 


©  2002  by  Carnegie  Mellon  University 


Carnegie  Mellon 

Software  Engineering  Institute 


Initial  Measurement  Areas 


Customer 


Customer  satisfaction  with  delivered  product 
Compliance  with  customer  requirements 
On  time  delivery 


Financial 


Internal  Business 


Funding  stability 
Trend  in  Expenses 


Availability  and  capability  of 
resources  (staff) 

Status  of  open  deficiencies  in 

delivered  projects 

Timeliness  of  projects  completion 


Innovation  &  Learning 

CMM  level 

Trend  in  employee  satisfaction 
Meeting  functional  requirements 
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Measurement  Areas  to  Indicators 


Indicators 
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Internal  Business 

Status  of  Open  Deficiencies  in  Delivered  Projects 


Number  of  Deficiencies 

That  Have  Been  Open  x  Days 

Severity 

Levels 

X  <  30 

30  <  X  >  60 

60  <  X  >  90 

X  >  90 

Totais 

Severity  1 

2 

1 

3 

Severity  2 

3 

1 

1 

5 

Severity  3 

3 

2 

1 

1 

7 

Severity  4 

4 

3 

3 

2 

12 

Severity  5 

8 

6 

3 

3 

20 

Totais 

20 

13 

8 

6 

47 
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Internal  Business 

Availability  and  Capability  of  Resources  (Staff) 


FY  99 

FY  00 

FY01 

# 

% 

# 

% 

# 

% 

GOAL 

Entry  Level 

E&S 

Journeyman 

40% 

High  Grade 

GOAL 

Entry  Level 

Tech 

Journeyman 

45% 

High  Grade 

GOAL 

Entry  Level 

Other 

Journeyman 

15% 

High  Grade 
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Internal  Business 

Timeliness  of  Project  Completion 


Completed  Projects  in  Reporting  Period 


on  time  or  early 

exceeded  original  schedule  by  less  than  10% 
exceeded  original  schedule  by  more  than  10% 
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Summary  of  Initial  Results 


Compliance  with  customer  requirements 


Total  Systems 

Full 

Compliance 

Partial 

Compliance 

# 

% 

# 

% 

>  Funding  stability 
Trend  in  Expenses 


Satisfied  with  delivered 
Product 

Compliant  with  requirements 
On-time  delivery 


Financial 

Customer 

Balanced 

Scorecard 

Internal 

Business 

Process 

Learning 

and 

Growth 

j 

Travel 


Avail.  &  capability  of  staff 
Status  of  open  deficiencies 
Timeliness  of  project  completion 


Trend  in  employee  satisfaction 
Meeting  functional  requirements 
CMM  Level 
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2  3  4  5 

SA/CMM  Level 


Number  of  Deficiencies 

That  Have  Been  Open  x  Days 

Severity 

Levels 

x<30 

30  <  X  >  60 

60  <  X  >  90 

x>90 

Totals 

Severity  1 

2 

1 

3 

Severity  2 

3 

1 

1 

5 

Severity  3 

3 

2 

1 

1 
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Severity  4 
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3 
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Severity  5 

8 
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3 

20 

Totals 

20 

13 

8 

6 
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Indicator 

Documentation 


Documents  the  why, 
what,  who,  when, 
where,  and  how 
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INDICATOR  TEMPLATE 

Measurement  Goal  # _ : 

Objective  - 

Questions  - 

Visual  Display 


Input(s) 

Data  Elements  — 

Responsibility  — 
for  Reporting 

Form(s)  ~ 
Algorithm  - 

Assumptions  - 

Interpretation  - 

X-reference 
Probing  Questions 
Evolution 
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Overview  Outline 

Methodology 

Major  components 

•  BSC 

•  GQ(I)M 

Example  use 

•  Initial  measurement  areas 

•  Indicators 

Summary 
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Summary 


The  approach,  using  the  BSC  and  GQ(I)M,  provides  a  systematic 
way  to  obtain  indicators  and  measures  that  reflect  the  health  and 
performance  of  the  organization. 

The  approach  uses  an  organization’s  vision  and  mission 
statements  to  identify  and  clarify  strategic  goals  and  sub-goals. 

The  sub-goals  are  mapped  to  the  balanced  scorecard. 

The  GQ(I)M  methodology  is  then  used  to  identify  measures  and 
indicators 


Bottom  Line 

We  tried  it;  It  worked;  Now  maturing  methodology 
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Mission 


Clarify  mission  & 
Vision  statement 


Sub-Goals 


draft  Vision 


Strategic  Goals 


Balanced  Scorecard 


Financial 
•  Sub-Goals 


Methodology 


Map  Sub-Goals  to  each 
quadrant  of  the 
Balanced  Score  Card 


Customer 

1  •  Sub-Goals 

Internal  Business 
•  Sub-Goals 

Learning  &  Growth 
•  Sub-Goals 

For  each  BSC  Quadrant 


Apply  GQ(I)M  to: 

-  identify  measurement  areas 

-  develop  measurement  goals 

-  pose  relevant  questions 

-  postulate  indicators 

-  identify  data  elements 


Indicators 


iIl  IIit: 


Data  Elements 
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Back-up  Material 
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Some  Definitions 


Performance  Management 

“The  use  of  performance  measurement  information  to  help 
set  agreed-upon  performance  goals,  allocate  and  prioritize 
resources,  inform  managers  to  either  confirm  or  change 
current  policy  or  program  directions  to  meet  those  goals, 
and  report  on  the  success  in  meeting  those  goals.” 

Performance  Measurement 
“A  process  of  assessing  progress  towards  achieving 
predetermined  goals,  including  information  on  [efficiency, 
quality,  and]  outcomes.... 


Source:  “Serving  the  American  Public:  Best  practices  in  performance  measurement,”  June  1997. 
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